From the retina to action: Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

This work was supported by ANR project "Horizontal-V1" N° ANR-17-CE37-0006".

https://laurentperrinet.github.io/talk/2019-04-03-a-course-on-vision-and-modelization

From the retina to action: Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

This work was supported by ANR project "Horizontal-V1" N° ANR-17-CE37-0006".

Churchland, P. S. & Sejnowski, T. J. (1992) The computational brain. Cambridge, MA: MIT Press.

From the retina to action: Levels of modeling

(see this viperlib page)

LP, Adams and Friston (2015) Biological Cybernetics

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment. It shows the states of the environment and the system in terms of a probabilistic dependency graph, where connections denote directed (causal) dependencies. The quantities are described within the nodes of this graph -- with exemplar forms for their dependencies on other variables.
Hidden (external) and internal states of the agent are separated by action and sensory states. Both action and internal states -- encoding a conditional probability density function over hidden states -- minimise free energy. Note that hidden states in the real world and the form of their dynamics can be different from that assumed by the generative model; (this is why hidden states are in bold. )

LP, Adams and Friston (2015) Biological Cybernetics

Outline

From the retina to action: Levels of modeling

Sparse coding in Neurophysiology and neural processing

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

Ravello, LP, Escobar, Palacios (2019) Scientific Reports

Kremkow, LP, Monier, Alonso, Aertsen, Fregnac, Masson (2016) Push-pull receptive field organization and synaptic depression: Mechanisms for reliably encoding naturalistic stimuli in V1

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

I wish to point here to an essential feature compared to classical feed-forward networks: you can not do the inference in one single shot in general; you need a recurrent / recursive network (see arrow) and this is precisely a possible function for one of the most numerous type of synapses: short-ranges lateral interactions

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

Olshausen and Field (1997) Sparse coding with an overcomplete basis set: A strategy employed by V1?

a seminal idea is proposed by Olshausen: * this may be formalized as an inference problem:

edges are different sources, which are known to be sparse: by mixing these sources one forms the image (transparency hypothesis)

the sparseness is characterized by the pdf of the sources coefficients

this inference problem is an inverse problem:
while this problem may be hard to solve, this may be approached using a (conjugate) gradient descent which has a nice implementation in terms of neural networks

LP (2010) Neural Computation

Outline

From the retina to action: Levels of modeling

Sparse coding in Neurophysiology and neural processing

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

      http://www.math.tamu.edu/~roquesol/Computational_Neuroscience_Summer_I_2018_Session_2-3_Print.pdf

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

/Users/laurentperrinet/research/OBV1/2018_HL_M1/figs

Outline

From the retina to action: Levels of modeling

Sparse coding in Neurophysiology and neural processing

Modeling Spiking Neural Networks

A model of Sparse Deep Predictive Coding

let's move to the second part : learning

the main goal of Olshasuen was not only sparse coding but the fact that using the sparse code, a simple linear hebbian learning allows to separate independent sources

one can go one step before the cortex and ask the same question in the retina are the same process present ?

Theoretical advances in neural networks modelling have recently been pushed by the field of machine learning. These have proposed that biological neural networks could process information according to certain constraints of efficiency, such as minimizing the surprise of sensory states given noisy or ambiguous sensory inputs. We have developed such a Hierarchical (Deep) Predictive Coding model using unsupervised learning on (static) natural images, and that we wish to extend to (dynamic) sensory flows. Compared to classical Deep Learning models, one advantage of this approach is that the learned synaptic weights are meaningful, that is that they often emerge as to represent explicitly components of the input image in increasing levels of complexity: edges at the first layer, curvatures in the second, the eye or mouth in the third layer when presenting images of faces. Such networks can also be extended to supervised or reinforcement learning by adding efferent layers that would accept such labels.

Boutin, Franciosini, Ruffier, LP (2019) submitted

(from http://cs231n.github.io/convolutional-networks/)

Boutin, Franciosini, Ruffier, LP (2019) submitted

Let's now apply that to natural images of faces

From the retina to action: Understanding visual processing

Laurent Perrinet

Licence Sciences & Humanité, 3/4/2019

This work was supported by ANR project "Horizontal-V1" N° ANR-17-CE37-0006".

The free parameters of this network are the anatomical constraints that we impose to dimension layers. In the particular case of this networks, this is the number of layers, the number of channels (that is the number of overlays of the same map in a given layer), and the size of the kernels implementing the convolutions (in resemblance to receptive fields in biological NNs). Indeed, as in most classical Deep Learning neural networks, these parameters are hand tuned. However, we observed that these superstructural numbers have a great influence on the quality and type of filters that could emerge from the learning phase. Theoretically, this make sense as it imposes different levels of competition within each layer across channels but also by defining a characteristic size in local interactions.

We believe that this theoretical problem is similar to that faced by different the primary visual cortex in different species. Given a similar sensory input, the intra-cortical network imposes different levels of anatomical complexities. We predict that, given realistic orders of magnitudes for different species, we could differentially predict the emergence of different macrostructures in different species. In particular, the emergence of a topographically smooth representation of orientations is present in marmosets while mice have a salt-and-pepper representation. Thanks to this level of understanding, we will use our models to make predictions. In particular, we can generate images which are optimal to evoke activity targeted at a given layer, for instance for the second layer (curvatures) versus the first layer (edges). In practice, such images take the form of textures with different levels of structural complexity. For instance, stimuli targeted at the first layer would correspond to a sub-class of stimuli that we have widely used in previous experiments (Motion Clouds). We predict that more complex classes of stimuli should evoke differential activities in both species.

- Task 5: Bluesky: “Efficient hierarchical representations ” Overall, the role of visual information processing in cortical layers is to represent information robustly to noise and visual transformations (translations zooms rotations). There is thus a pressure on the structure of this processing at different temporal scales to maximize the efficiency of this code. This depends both on the statistics of the sensory input (both in space and time) but also on the repertoire of behavioral tasks that are embodied. We have recently developed a hierarchical unsupervised algorithm for neural networks which allows to generate such representations and that fits well to the processing performed by feed-forward, lateral and feedback connections. In such a model, the message passing between neurons is implemented by propagating waves which progressively refine the representation of sensory data. In particular, tuning the complexity of the representation at different layers allows to see the emergence of different structures. In particular, progressively increasing the complexity in lateral connections shows a phase transition from a salt-and-pepper organization to a more continuous representation of the space x orientation space. This could explain the different architectures observed in different animals. Such models could also make predictions on the adaptation of the network to novel environmental or behavioral contingencies. The goal of this Task is two-sided: first to understand the physiological and behavioral data in the light of predictive coding theories, rooting these results in a quantitative model. The second goal is to generate predictions. In particular, using our models, it will be possible to generate novel stimulus classes that specifically tackle the efficiency of the visual system at its different spatial and dynamic granularities. Lead Laurent/Lyle?

Thanks for your attention!

https://laurentperrinet.github.io/talk/2019-04-03-a-course-on-vision-and-modelization